Systems and methods of sharing an augmented environment with a companion

ABSTRACT

A system includes one or more memory devices storing instructions, and one or more processors configured to execute the instructions to perform steps of a method for providing an interactive virtual environment. The system may receive environmental data, and augment the environmental data by adding virtual environmental data. The system may then provide this virtual environment to two or more devices. The system can then receive user input, and modify the virtual environment in response to those inputs.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/664,705, filed 31 July 2017, which claims the benefit of U.S.Provisional Application No. 62/437,697, filed 22 December 2016, theentire contents and substance of which are hereby incorporated byreference as if fully set forth below.

FIELD OF INVENTION

The present disclosure relates to systems and methods for providing twoor more users with an augmented environment, and more particularly aninteractive virtually augmented environment.

BACKGROUND

Certain experiences are better when shared with one or more otherpeople, but having someone else physically present is not alwayspossible. For example, a shopper may wish to have a companion helpselect product such as clothing, home goods, or groceries. In the eventthat another person is unable to physically be present, video chatapplications are currently available to provide two-way audiovisualcommunication using one or more cameras and/or microphones. Theseapplications can allow a person not physically present to see and hearsome of the experience.

However, these applications cannot provide the users with a truly sharedenvironment, and the remote individual often lacks the ability toclearly indicate or act upon items in the experience. For example, if auser is shopping for clothes, the remote individual has no ability tomove, change, or directly indicate anything in the shopper's vision. Theremote individual has no way of, for example, handing the shopper anitem of clothing to try on, or even indicating a particular item on ashelf or display. While the current applications allow some limitedinteraction, they do not provide a truly shared and interactiveenvironment.

Accordingly, there is a need for systems and methods that allow a deviceto collect information about an experience, augment that informationwith interactive virtual elements, and share that augmented environmentbetween two or more people. Embodiments of the present disclosure aredirected to this and other considerations.

SUMMARY

Disclosed embodiments provide systems and methods for sharing anaugmented environment with a companion.

Consistent with the disclosed embodiments, the system may include one ormore memory devices storing instructions, and one or more processorsconfigured to execute the instructions to perform steps of a method toprovide an interactive virtual environment. The system may execute theinstructions to receive environmental data comprising audio and visualinformation representing a physical environment from a first computingdevice. The system may then augment the environmental data by addingvirtual environmental data comprising data associated with a merchant.The one or more processors may then provide, to the first computingdevice, the virtual environmental data to create an augmentedenvironment in combination with the physical environment. The system maythen provide the environmental data and the virtual environmental datato create a virtual environment to a second computing device. The systemmay then receive user input data associated with one or more predefineduser inputs from the second computing device. In response to the userinput data, the system can then modify the virtual environment on thesecond computing device and the augmented environment on the firstcomputing device to display modified virtual environmental data based onthe user input data.

Consistent with the disclosed embodiments, methods for providing aninteractive virtual environment are also disclosed.

Further features of the disclosed design, and the advantages offeredthereby, are explained in greater detail hereinafter with reference tospecific embodiments illustrated in the accompanying drawings, whereinlike elements are indicated by like reference designators.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and which are incorporated into andconstitute a portion of this disclosure, illustrate variousimplementations and aspects of the disclosed technology and, togetherwith the description, serve to explain the principles of the disclosedtechnology. In the drawings:

FIG. 1 is a diagram of an exemplary system that may be used to generateand share a virtual environment;

FIG. 2 is a component diagram of an exemplary service provider terminal;

FIG. 3 is a flowchart of an exemplary method for generating and sharinga virtual environment;

FIG. 4 is a flowchart of an exemplary method for generating and sharinga virtual environment;

FIG. 5 is a diagram of exemplary types of virtual environmental data;

FIG. 6 is an exemplary display of an embodiment of a computing device;and

FIG. 7 is a component diagram of an exemplary computing device.

DETAILED DESCRIPTION

Some implementations of the disclosed technology will be described morefully with reference to the accompanying drawings. This disclosedtechnology may, however, be embodied in many different forms and shouldnot be construed as limited to the implementations set forth herein. Thecomponents described hereinafter as making up various elements of thedisclosed technology are intended to be illustrative and notrestrictive. Many suitable components that would perform the same orsimilar functions as components described herein are intended to beembraced within the scope of the disclosed electronic devices andmethods. Such other components not described herein may include, but arenot limited to, for example, components developed after development ofthe disclosed technology.

It is also to be understood that the mention of one or more method stepsdoes not preclude the presence of additional method steps or interveningmethod steps between those steps expressly identified. Similarly, it isalso to be understood that the mention of one or more components in adevice or system does not preclude the presence of additional componentsor intervening components between those components expressly identified.

The disclosed embodiments are directed to systems and methods forsharing an interactive augmented environment with a companion. Thesystem may include one or more memory devices storing instructions, andone or more processors configured to execute the instructions to performsteps of a method. Specifically, in some embodiments, the system mayprovide an interactive virtual environment. To accomplish this, thesystem may execute the instructions to receive environmental datacomprising audio and visual information representing a physicalenvironment from a first computing device. The system may then augmentthe environmental data by adding virtual environmental data comprisingdata associated with a merchant. The one or more processors may thenprovide, to the first computing device, the virtual environmental datato create an augmented environment in combination with the physicalenvironment. The system may then create a virtual environment byproviding the environmental data and the virtual environmental data to asecond computing device. The system may then receive user input dataassociated with one or more predefined user inputs from the secondcomputing device. In response to the user input data, the system canthen modify the virtual environment on the second computing device andthe augmented environment on the first computing device to displaymodified virtual environmental data based on the user input data.

In another embodiment, a system for providing an interactive augmentedenvironment that may be used to create a shared experience between usersof two computing devices may include an environmental data sensor forobtaining environmental data comprising audio and/or visual informationrepresenting a physical environment. The system may include aninput/output device for transmitting environmental data to a firstcomputing device, receiving virtual environmental data comprising dataassociated with a merchant from the first computing device, andreceiving an user input data associated with one or more predefined userinputs from the first computing device. The system may further includean augmented environment processor that, based on environmental dataobtained from the environmental data sensor and virtual environmentaldata received from the first computing device, generates a signalcomprising augmented environmental data, and modifies the signalcomprising augmented environmental data to integrate enhanced contentbased on user input data received from the first computing device. Theenhanced content may represent content that is unrepresented in thephysical environment. The system may include an augmented environmentaldata display that receives the signal comprising augmented environmentaldata and displays said augmented environmental data in accordance withsaid signal.

In another aspect, a non-transitory computer-readable medium storinginstructions is disclosed. The instructions, when executed by one ormore processors, may cause a computing device to perform steps of amethod for providing an interactive virtual environment. Specifically,the computing device may receive environmental data comprising audio andvisual information representing a physical environment from a firstcomputing device. The computing device may then determine whether any ofthe purchased products on the listing correspond to one or morepredetermined complementary product sets. This determination may resultin the identification of one or more unpurchased products from the oneor more complementary product sets. The instructions may then cause thecomputing device to augment the environmental data by adding virtualenvironmental data comprising data associated with a merchant.

In yet another respect, a method for providing an interactive virtualenvironment is disclosed. The method may include providing one or morememory devices storing instructions, and one or more processorsconfigured to execute those instructions. The instructions may cause thesystem to provide, to the first computing device, the virtualenvironmental data to create an augmented environment in combinationwith the physical environment, and provide the environmental data andthe virtual environmental data to create a virtual environment to asecond computing device. After providing the virtual environmental data,the system may receive user input data associated with one or morepredefined user inputs from the second computing device. The system maythen modify the virtual environment on the second computing device andthe augmented environment on the first computing device to displaymodified virtual environmental data based on the user input data.

Reference will now be made in detail to exemplary embodiments of thedisclosed technology, examples of which are illustrated in theaccompanying drawings and disclosed herein. Wherever convenient, thesame references numbers will be used throughout the drawings to refer tothe same or like parts.

FIG. 1 is a diagram of an exemplary system that may be configured toperform one or more processes that can generate interactive virtualelements to augment an environment. The components and arrangementsshown in FIG. 1 are not intended to limit the disclosed invention as thecomponents used to implement the disclosed processes and features mayvary.

In accordance with disclosed embodiments, an interactivevirtual/augmented environment system 100 may include a service providerterminal 110 in communication with a first computing device 120 and asecond computing device 130 via network 140. First and second computingdevices 120 and 130 may include a mobile computing device (e.g., a smartphone, tablet computer, smart wearable device, portable laptop computer,voice command device, or other mobile computing device) or a stationarycomputing device (e.g., a stationary desktop or other stationarycomputer). Computing devices 120 and 130 may belong to or be provided bya consumer, or may be borrowed, rented, or shared. In some embodiments,computing devices 120 and 130 may be configured to provide one or moreof an augmented reality and a virtual reality experience to one or moreusers, and may be equipped with a display, speakers or other auditorydevices, tactile simulators, haptic sensors, cameras, light projectors,input devices, distance measuring equipment, 3D scanners, IR sensors,microphones, orientation/position/location sensors, accelerometers, andother devices configured to engage the user's senses to enhance theaugmented/virtual reality experience.

In some embodiments, interactive virtual/augmented environment system100 may also include a merchant database terminal 150 that communicateswith other devices of system 100 via network 140. Optionally, in someembodiments, interactive virtual/augmented environment system 100 mayalso include a merchant point of sale (POS) terminal 160 thatcommunicates with other devices of system 100 via network 140.Additionally, in some embodiments, interactive virtual/augmentedenvironment system 100 may also include a financial service providerterminal 170 that communicates with other devices of system 100 vianetwork 140.

Network 140 may be of any suitable type, including individualconnections via the internet such as cellular or WiFi networks. In someembodiments, network 140 may connect terminals using direct connectionssuch as radio-frequency identification (RFID), near-field communication(NFC), Bluetooth™, low-energy Bluetooth™ (BLE), WiFi™, ZigBee™, Z-Wave™,ambient backscatter communications (ABC) protocols, USB, or LAN. Becausethe information transmitted may be personal or confidential, securityconcerns may dictate one or more of these types of connections beencrypted or otherwise secured. In some embodiments, however, theinformation being transmitted may be less personal, and therefore thenetwork connections may be selected for convenience over security.

An exemplary embodiment of service provider terminal 110 is shown inmore detail in FIG. 2. Computing devices 120 and 130, merchant databaseterminal 150, merchant POS terminal 160, and financial service providerterminal 170 may have a similar structure and components that aresimilar to those described with respect to service provider terminal110. As shown, service provider terminal 110 may include a processor210, an input/output (“I/O”) device 220, a memory 230 containing anoperating system (“OS”) 240 and a program 250. For example, serviceprovider terminal 110 may be a single server or may be configured as adistributed computer system including multiple servers or computers thatinteroperate to perform one or more of the processes and functionalitiesassociated with the disclosed embodiments. In some embodiments, theservice provider terminal 110 may further include a peripheralinterface, a transceiver, a mobile network interface in communicationwith the processor 210, a bus configured to facilitate communicationbetween the various components of the service provider terminal 110, anda power source configured to power one or more components of the serviceprovider terminal 110.

A peripheral interface may include the hardware, firmware and/orsoftware that enables communication with various peripheral devices,such as media drives (e.g., magnetic disk, solid state, or optical diskdrives), other processing devices, or any other input source used inconnection with the instant techniques. In some embodiments, aperipheral interface may include a serial port, a parallel port, ageneral purpose input and output (GPIO) port, a game port, a universalserial bus (USB), a micro-USB port, a high definition multimedia (HDMI)port, a video port, an audio port, a Bluetooth™ port, a near-fieldcommunication (NFC) port, another like communication interface, or anycombination thereof.

In some embodiments, a transceiver may be configured to communicate withcompatible devices and ID tags when they are within a predeterminedrange. A transceiver may be compatible with, for example, one or moreof: radio-frequency identification (RFID), near-field communication(NFC), Bluetooth™, low-energy Bluetooth™ (BLE), WiFi™, ZigBee™, Z-Wave™,ambient backscatter communications (ABC) protocols or similartechnologies.

A mobile network interface may provide access to a cellular network, theInternet, or another wide-area network. In some embodiments, a mobilenetwork interface may include hardware, firmware, and/or software thatallows the processor(s) 210 to communicate with other devices via wiredor wireless networks, whether local or wide area, private or public, asknown in the art. A power source may be configured to provide anappropriate alternating current (AC) or direct current (DC) to powercomponents.

Processor 210 may include one or more of a microprocessor,microcontroller, digital signal processor, co-processor or the like orcombinations thereof capable of executing stored instructions andoperating upon stored data. Memory 230 may include, in someimplementations, one or more suitable types of memory (e.g. such asvolatile or non-volatile memory, random access memory (RAM), read onlymemory (ROM), programmable read-only memory (PROM), erasableprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), magnetic disks, optical disks,floppy disks, hard disks, removable cartridges, flash memory, aredundant array of independent disks (RAID), and the like), for storingfiles including an operating system, application programs (including,for example, a web browser application, a widget or gadget engine, andor other applications, as necessary), executable instructions and data.In one embodiment, the processing techniques described herein areimplemented as a combination of executable instructions and data withinthe memory 230.

Processor 210 may be one or more known processing devices, such as, butnot limited to, a microprocessor from the Pentium™ family manufacturedby Intel™ or the Turion™ family manufactured by AMD™. Processor 210 mayconstitute a single core or multiple core processor that executesparallel processes simultaneously. For example, processor 210 may be asingle core processor that is configured with virtual processingtechnologies. In certain embodiments, processor 210 may use logicalprocessors to simultaneously execute and control multiple processes.Processor 210 may implement virtual machine technologies, or othersimilar known technologies to provide the ability to execute, control,run, manipulate, store, etc. multiple software processes, applications,programs, etc. One of ordinary skill in the art would understand thatother types of processor arrangements could be implemented that providefor the capabilities disclosed herein.

Service provider terminal 110 may include one or more storage devicesconfigured to store information used by processor 210 (or othercomponents) to perform certain functions related to the disclosedembodiments. In one example, service provider terminal 110 may includememory 230 that includes instructions to enable processor 210 to executeone or more applications, such as server applications, networkcommunication processes, and any other type of application or softwareknown to be available on computer systems. Alternatively, theinstructions, application programs, etc. may be stored in an externalstorage or available from a memory over a network. The one or morestorage devices may be a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, or other type ofstorage device or tangible computer-readable medium.

In one embodiment, service provider terminal 110 may include memory 230that includes instructions that, when executed by processor 210, performone or more processes consistent with the functionalities disclosedherein. Methods, systems, and articles of manufacture consistent withdisclosed embodiments are not limited to separate programs or computersconfigured to perform dedicated tasks. For example, service providerterminal 110 may include memory 230 that may include one or moreprograms 250 to perform one or more functions of the disclosedembodiments. Moreover, processor 210 may execute one or more programs250 located remotely from system 100. For example, system 100 may accessone or more remote programs 250, that, when executed, perform functionsrelated to disclosed embodiments.

Memory 230 may include one or more memory devices that store data andinstructions used to perform one or more features of the disclosedembodiments. Memory 230 may also include any combination of one or moredatabases controlled by memory controller devices (e.g., server(s),etc.) or software, such as document management systems, Microsoft™ SQLdatabases, SharePoint™ databases, Oracle™ databases, Sybase™ databases,or other relational databases. Memory 230 may include softwarecomponents that, when executed by processor 210, perform one or moreprocesses consistent with the disclosed embodiments. In someembodiments, memory 230 may include a merchant information database 260,an environment database 270, and a virtual environmental database 280for storing related data to enable service provider terminal 110 toperform one or more of the processes and functionalities associated withthe disclosed embodiments. Merchant information database 260 may includestored data relating to product price, product inventory, relatedproducts, and available discounts associated with one or more productsassociated with one or more merchants. Environment database 270 mayinclude stored data relating to a physical environment that is capturedby one or more devices (e.g., first computing device 120) of system 100.Virtual environment database 280 may include stored data relating to avirtual environment that is provided to and can be updated based uponinteractions with one or more devices (e.g., computing devices 120 and130) of system 100.

Service provider terminal 110 may also be communicatively connected toone or more memory devices (e.g., databases (not shown)) locally orthrough a network. The remote memory devices may be configured to storeinformation and may be accessed and/or managed by service providerterminal 110. By way of example, the remote memory devices may bedocument management systems, Microsoft™ SQL databases, SharePoint™databases, Oracle™ databases, Sybase™ databases, or other relationaldatabases. Systems and methods consistent with disclosed embodiments,however, are not limited to separate databases or even to the use of adatabase.

Service provider terminal 110 may also include one or more I/O devices220 that may comprise one or more interfaces for receiving signals orinput from devices and providing signals or output to one or moredevices that allow data to be received and/or transmitted by serviceprovider terminal 110. For example, service provider terminal 110 mayinclude interface components, which may provide interfaces to one ormore input devices, such as one or more keyboards, mouse devices, touchscreens, track pads, trackballs, scroll wheels, digital cameras,microphones, sensors, and the like, that enable service providerterminal 110 to receive data from one or more users (such as via userterminal 120).

In exemplary embodiments of the disclosed technology, service providerterminal 110 may include any number of hardware and/or softwareapplications that are executed to facilitate any of the operations. Theone or more I/O interfaces may be utilized to receive or collect dataand/or user instructions from a wide variety of input devices. Receiveddata may be processed by one or more computer processors as desired invarious implementations of the disclosed technology and/or stored in oneor more memory devices.

While service provider terminal 110 has been described as one form forimplementing the techniques described herein, those having ordinaryskill in the art will appreciate that other, functionally equivalenttechniques may be employed. For example, as known in the art, some orall of the functionality implemented via executable instructions mayalso be implemented using firmware and/or hardware devices such asapplication specific integrated circuits (ASICs), programmable logicarrays, state machines, etc. Furthermore, other implementations of theterminal 110 may include a greater or lesser number of components thanthose illustrated.

Merchant database terminal 150 may have one or more processors 152 andone or more merchant databases 154, which may be any suitable repositoryof merchant data. Merchant database terminal 150 may be located at thePOS location, off-site at another merchant location, or at a third partylocation. Information stored in merchant database 154 may be accessed(e.g., retrieved, updated, and added to) via network 140 by one or moredevices (e.g., service provider terminal 110) of system 100. Merchantdatabase 154 may store information relating to products and servicesoffered by merchants such as pricing, quantity, availability, discounts,reviews, and any other such generally available information that aconsumer may utilize in making a purchasing decision. In someembodiments, merchant database 154 may also include location informationassociated with products and services that identifies the location(s)that a particular product or service is available for purchase. In someembodiments, the location information may include an identification of aparticular store, terminal, or kiosk that the product or service may bepurchased from.

Merchant POS terminal 160 may have one or more POS devices 162, 164, 166that communicate with one or more devices (e.g., computing devices 120and 130) of system 100 via network 140. In some embodiments, POS devices162, 164, 166 may be associated with one or more products or items at aPOS location, and may serve as beacons to identify the associatedproducts or items and related information to one or more devices ofsystem 100.

Financial service provider terminal 170 may have one or more processors172 and one or more financial service provider databases 174, which maybe any suitable repository of merchant data. Financial service providerterminal 170 may be located at the POS location, off-site at anotherfinancial service provider location, or at a third party location.Information stored in financial service provider database 174 may berequested and sent to one or more devices (e.g., service providerterminal 110, merchant POS terminal 160) of system 100 via network 140.

FIG. 7 shows an exemplary embodiment of first computing device 120. Asshown, first computing device 120 may include input/output (“I/O”)device 220 for transmitting and receiving data to and from anothercomputing device (such as service provider terminal 110), memory 230containing operating system (“OS”) 240 and program 250 and allassociated components as described above with respect to serviceprovider terminal 110. First computing device 120 may also have one ormore processors, including an augmented environment processor 702 forgenerating signals representative of augmented and/or virtualenvironments, a geographic location sensor (“GLS”) 704 for determiningthe geographic location of first computing device 120 and/or anassociated environmental data sensor 708, an augmented environmentaldisplay 706 for displaying augmented environmental data, and anenvironmental data (“ED”) sensor 708 for obtaining environmental dataincluding audio and/or visual information representing a physicalenvironment. In some embodiments, environmental data sensor 708 mayinclude a microphone or a digital camera. According to some embodiments,augmented environment processor 702 may include all of the features andfunctions of processor 210 described above. Further, augmentedenvironment processor 702 may be configured to generate a signalcomprising augmented environmental data based on environmental dataobtained from the environmental data sensor 708. The augmentedenvironment processor 702 may also be configured to modify the signalcomprising augmented environmental data to integrate enhanced contentbased on user input received from a computing device (such as serviceprovider terminal 110). The enhanced content may be content that isunrepresented in the physical environment. In some embodiments, theaugmented environment processor 702 may identify one or more products inthe visual information obtained by the environmental data sensor 708 andmay associate merchant data from a merchant database 154 with one ormore identified products.

In some embodiments, an augmented environment data signal may be used tocreate an augmented reality display of the physical environmentincluding enhanced content. Further, another device (e.g., secondcomputing device 130) may utilize the environmental data and virtualenvironmental data to create a virtual reality display of the physicalenvironment including enhanced content. Accordingly, system 100 mayenable the simultaneously display of an augmented reality display on afirst computing device 120 and a virtual reality display of the samephysical location on a second computing device 130, so that the users ofthe two devices may have a shared experience. In some embodiments,merchant data associated with the physical environment may beincorporated into either of the augmented reality display or the virtualreality display. Although FIG. 7 is described with respect to a firstcomputing device 120, it is contemplated that in some embodiments,second computing device 130 may include some or all of the elementsdisclosed with respect to FIG. 7.

FIG. 3 shows a flowchart of method 300 for generating and sharing anaugmented environment. Method 300 may be performed by service providerterminal 110 using processor 210 to execute memory 230. In someembodiments, steps of method 300 may be delegated to other elements insystem 100, such as the computing devices 120 and 130 and/or merchantdatabase terminal 150. Following method 300, the users may generate,share, and modify virtually augmented environments.

In block 310, service provider terminal 110 may receive environmentaldata comprising audio and visual information via first computing device120. At step 320, service provider terminal 110 may augment theenvironmental data by adding virtual environmental data. For example,this augmentation may involve identifying one or more products in thevisual information and displaying information from the merchant that isspecific to one of the one or more of the identified products. Productsor other items can be identified by recognizing shapes, bar codes, QRcodes, IR beacons or Bluetooth™ beacons located in an environment, orthe like. Products may be identified by matching the visual informationto a merchant-specific or non-merchant specific product database.Additionally, the system may pull in data from a synced user profile oruser device to populate the augmented environment. Once those items areidentified, service provider terminal 110 can connect to an appropriatemerchant database terminal 150 to provide information relevant to theidentified product or item.

In block 330, service provider terminal 110 may provide the virtualenvironmental data to first computing device 120 in order to create anaugmented environment when viewed in combination with the physicalenvironment. In some embodiments, the virtual environment data mayinclude some or all of the environmental data provided by firstcomputing device 120. In some embodiments, first computing device 120may only receive the newly generated data. The virtual environmentaldata may include product price information, product inventoryinformation, product manufacturing information, product marketinginformation or description, product ratings or reviews, socialinformation (e.g. if friend has bought/pinned/added to a wishlist),related products, available discounts, and aesthetic visual effects(e.g., target graphics, glow, highlighted colors, or firework graphicsdisplayed around one or more products). For example, service providerterminal may recognize a particular item or product and augment theenvironment with visual effects such as a price tag, other availablebrands or sizes, or aesthetic effects such as virtual fireworks or salesigns. Since second computing device 130 is not present to collect theenvironmental data, at step 340 that data may be provided along with thevirtual environmental data to allow second computing device 130 todisplay or provide the augmented environment to a user.

After displaying the augmented environment to a user, second computingdevice 130 may receive user input in response to that augmentedenvironment (block 350). For example, the user of second computingdevice 130 may touch a screen, make a gesture in the visual field of avideo capturing component of second computing device 130, speak acommand into an audio capturing component of second computing device130, enter an input using a connected input device, or some combinationof these. In some embodiments, second computing device 130 may be avirtual reality device including a headset and a camera. In suchembodiments, the user may be viewing the augmented environment, and mayphysically reach for an element in that environment to generate userinput data. User input data may also be generated by the user'sinteraction with keyboards, mouse devices, touch screens, track pads,trackballs, scroll wheels, digital cameras, microphones, sensors, andthe like. In some embodiments, second computing device 130 can be apersonal computer, with the augmented environment displayed on amonitor. A user may then select (e.g., using the mouse or othercontroller, or via motion if a motion sensor or device with anaccelerometer is used in cooperation with the computer) a virtualelement in the augmented environment to generate user input data.

After receiving the user input data, at block 360 service providerterminal 110 may modify the virtual environment in response to thisinput, and provide the modified augmented environment to first computingdevice 120, and optionally also to second computing device 130. Anexample of this modification may include moving a virtual element,changing a virtual element, adding or subtracting a virtual element, orany similar modification. The user of first computing device 120 canthen perceive the virtual element changing or moving in the real worldenvironment, via the augmented environment. In this way, the two userscan not only share audio and visual information as in a video chatapplication, but can also manipulate virtual elements such as productprice tags, aesthetic visual effects, or move items to a virtualshopping cart or return rack.

Optionally, in some embodiments, service provider terminal 110 mayreceive user input via one or more of computing devices 120 and 130indicating a desire to purchase one or more identified products. Inresponse, service provider terminal 110 transmit a purchaseauthorization request to financial service provider terminal 170, whichmay already have user account information associated with first and/orsecond computing devices 120 and 130. Once the request has beenauthorized, service provider terminal 100 may transmit a purchaserequest to merchant POS terminal 160 to purchase the identifiedproducts. Alternatively, in other embodiments, service provider terminal100 may add the identified products to a checkout cart associated withfirst and/or second computing devices 120 and 130 for purchase atanother time.

FIG. 4 shows a flowchart of method 400 for generating and sharing anaugmented environment wherein both the first and second computingdevices 120 and 130 can receive environmental data directly. Method 400may be performed by service provider terminal 110 using processor 210 toexecute memory 230. In some embodiments, steps of method 400 may bedelegated to other elements in system 100, such as the computing devices120 and 130 and/or merchant database terminal 150. Following method 400,the users may generate, share, and modify virtually augmentedenvironments.

In block 410, service provider terminal 110 may receive environmentaldata comprising audio and visual information via the first computingdevice 120 as well as the second computing device 130. At step 420,service provider terminal 110 may augment the environmental data byadding virtual environmental data in a manner similar to that discussedabove with respect to step 320.

In block 430, service provider terminal 110 may provide the virtualenvironmental data to the first and second computing devices 120 and 130in order to create an augmented environment when viewed in combinationwith the physical environment. Since the first and second computingdevices 120 and 130 may not be collecting the exact same environmentaldata, at step 430 the virtual environmental data may be overlaid suchthat virtual elements are related to locations in space to provide theaugmented environment to a user. For example, the first and secondcomputing devices may be in the same store, but in different aisles. Insuch a situation, the augmented environments created by each device mayinclude different virtual elements at any given time, however bothdevices can generate the same augmented environments when in the samelocation. When both devices are viewing the same shelf in the sameaisle, they can each include a virtual element displaying merchantinformation. Modifications to the augmented environment may not beimmediately viewed by both devices, but those modifications can bepresent at a particular location in the physical environment.

After displaying the augmented environment to the users, first andsecond computing devices 120 and 130 may receive user inputs in responseto the augmented environment (block 440). For example, the user of thefirst or second computing devices 120 or 130 may touch a screen, make agesture in the visual field of a video capturing component of thecomputing device, speak a command into an audio capturing component ofthe computing device, enter an input using a connected input device, orsome combination of these. After receiving the user input data at block440, at block 450, service provider terminal 110 may modify the virtualenvironmental data in response to this input, and provide the modifiedenvironmental data to the first and second computing devices 120 and130. In some embodiments these modifications may include moving avirtual element, changing a virtual element, adding or subtracting avirtual element, or any similar modification.

For example, service provider terminal 110 may receive user input datafrom first computing device 120 requesting a virtual marker be placednext to a particular item on a store shelf. Once that shelf locationbecomes present in the environmental data of second computing device130, the virtual marker will be present in response to the modificationto the virtual environmental data in response to the user input datafrom first computing device 120. In this way, modifications made to thevirtual environment by either computing device may be present in theaugmented environment present on both devices.

FIG. 5 shows a diagram that represents an embodiment of exemplary typesof virtual environmental data 500. For example, virtual environmentaldata 500 may include product price information 510, product inventoryinformation 520, product manufacturing information 530, productmarketing information or description 540, product ratings or reviews550, social information 560, related products 570, available discounts580, and particular visual effects 590.

FIG. 6 shows an exemplary display of an embodiment of first computingdevice 120. As shown, augmented environment 600, which may include, forexample, currently selected item 620, shopping cart 630, status window640, companion window 650 and the like, is overlaid on environment 610so that a user can interact with both real and virtual elements.Currently selected item 620 may include images or 3D models of an itemor items that can be manipulated to allow the item to be tried on,purchased, added to cart, returned to the shelf, or the like.Interactions with currently selected item 620 may also allow the user totoggle views, sizes, colors, styles, or other productvariations/information.

Augmented environment 600 can also include shopping cart feature 630.Shopping cart 630 may display a list of items selected to be stored forlater check out. Items can be added to the shopping cart 630 by thepresent user, the remote user, via a web interface before or during theshopping experience, or any combination of users or methods asappropriate. Augmented environment 600 can also include a status window640 that can give information on the shopping experience, sales,location, or other characteristics useful to the user. In someembodiments, status window 640 can be movable so as to not obstructparticular items of interest.

The user of second computing device 130 may be shown in companion window650 to allow the users to communicate and share the experience. As withother elements of augmented environment 600, companion window 650 may bemovable or resizable to customize the shopping experience. In someembodiments, multiple companion windows 650 may be employed to allowadditional users to share the experience. Various configurations ofaugmented environment 600 may be used, including unique environments fordifferent stores, departments, and/or times of year (such as seasonalenvironments, environments based on a particular sale or event, or thelike). Service provider terminal 110 may store individual userpreferences for the configuration of augmented environment 600.

As used in this application, the terms “component,” “module,” “system”and the like are intended to include a computer-related entity, such asbut not limited to hardware, firmware, a combination of hardware andsoftware, software, or software in execution. For example, a componentmay be, but is not limited to being, a process running on a processor,an object, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on acomputing device and the computing device can be a component. One ormore components can reside within a process and/or thread of executionand a component may be localized on one computer and/or distributedbetween two or more computers. In addition, these components can executefrom various computer readable media having various data structuresstored thereon. The components may communicate by way of local and/orremote processes such as in accordance with a signal having one or moredata packets, such as data from one component interacting with anothercomponent in a local system, distributed system, and/or across a networksuch as the Internet with other systems by way of the signal.

Certain embodiments and implementations of the disclosed technology aredescribed above with reference to block and flow diagrams of systems andmethods and/or computer program products according to exampleembodiments or implementations of the disclosed technology. It will beunderstood that one or more blocks of the block diagrams and flowdiagrams, and combinations of blocks in the block diagrams and flowdiagrams, respectively, can be implemented by computer-executableprogram instructions. Likewise, some blocks of the block diagrams andflow diagrams may not necessarily need to be performed in the orderpresented, may be repeated, or may not necessarily need to be performedat all, according to some embodiments or implementations of thedisclosed technology.

These computer-executable program instructions may be loaded onto ageneral-purpose computer, a special-purpose computer, a processor, orother programmable data processing apparatus to produce a particularmachine, such that the instructions that execute on the computer,processor, or other programmable data processing apparatus create meansfor implementing one or more functions specified in the flow diagramblock or blocks. These computer program instructions may also be storedin a computer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meansthat implement one or more functions specified in the flow diagram blockor blocks.

As an example, embodiments or implementations of the disclosedtechnology may provide for a computer program product, including acomputer-usable medium having a computer-readable program code orprogram instructions embodied therein, said computer-readable programcode adapted to be executed to implement one or more functions specifiedin the flow diagram block or blocks. Likewise, the computer programinstructions may be loaded onto a computer or other programmable dataprocessing apparatus to cause a series of operational elements or stepsto be performed on the computer or other programmable apparatus toproduce a computer-implemented process such that the instructions thatexecute on the computer or other programmable apparatus provide elementsor steps for implementing the functions specified in the flow diagramblock or blocks.

Accordingly, blocks of the block diagrams and flow diagrams supportcombinations of means for performing the specified functions,combinations of elements or steps for performing the specifiedfunctions, and program instruction means for performing the specifiedfunctions. It will also be understood that each block of the blockdiagrams and flow diagrams, and combinations of blocks in the blockdiagrams and flow diagrams, can be implemented by special-purpose,hardware-based computer systems that perform the specified functions,elements or steps, or combinations of special-purpose hardware andcomputer instructions.

Certain implementations of the disclosed technology are described abovewith reference to mobile computing devices. Those skilled in the artrecognize that there are several categories of mobile devices, generallyknown as portable computing devices that can run on batteries but arenot usually classified as laptops. For example, mobile devices caninclude, but are not limited to portable computers, tablet PCs, internettablets, PDAs, ultra mobile PCs (UMPCs), wearable devices, and smartphones. Additionally, implementations of the disclosed technology can beutilized with internet of things (IoT) devices, smart televisions andmedia devices, appliances, automobiles, toys, and voice command devices,along with peripherals that interface with these devices.

In this description, numerous specific details have been set forth. Itis to be understood, however, that implementations of the disclosedtechnology may be practiced without these specific details. In otherinstances, well-known methods, structures and techniques have not beenshown in detail in order not to obscure an understanding of thisdescription. References to “one embodiment,” “an embodiment,” “someembodiments,” “example embodiment,” “various embodiments,” “oneimplementation,” “an implementation,” “example implementation,” “variousimplementations,” “some implementations,” etc., indicate that theimplementation(s) of the disclosed technology so described may include aparticular feature, structure, or characteristic, but not everyimplementation necessarily includes the particular feature, structure,or characteristic. Further, repeated use of the phrase “in oneimplementation” does not necessarily refer to the same implementation,although it may.

Throughout the specification and the claims, the following terms take atleast the meanings explicitly associated herein, unless the contextclearly dictates otherwise. The term “connected” means that onefunction, feature, structure, or characteristic is directly joined to orin communication with another function, feature, structure, orcharacteristic. The term “coupled” means that one function, feature,structure, or characteristic is directly or indirectly joined to or incommunication with another function, feature, structure, orcharacteristic. The term “or” is intended to mean an inclusive “or.”Further, the terms “a,” “an,” and “the” are intended to mean one or moreunless specified otherwise or clear from the context to be directed to asingular form. By “comprising” or “containing” or “including” is meantthat at least the named element, or method step is present in article ormethod, but does not exclude the presence of other elements or methodsteps, even if the other such elements or method steps have the samefunction as what is named.

While certain embodiments of this disclosure have been described inconnection with what is presently considered to be the most practicaland various embodiments, it is to be understood that this disclosure isnot to be limited to the disclosed embodiments, but on the contrary, isintended to cover various modifications and equivalent arrangementsincluded within the scope of the appended claims. Although specificterms are employed herein, they are used in a generic and descriptivesense only and not for purposes of limitation.

This written description uses examples to disclose certain embodimentsof the technology and also to enable any person skilled in the art topractice certain embodiments of this technology, including making andusing any apparatuses or systems and performing any incorporatedmethods. The patentable scope of certain embodiments of the technologyis defined in the claims, and may include other examples that occur tothose skilled in the art. Such other examples are intended to be withinthe scope of the claims if they have structural elements that do notdiffer from the literal language of the claims, or if they includeequivalent structural elements with insubstantial differences from theliteral language of the claims.

Exemplary Use Cases

The following exemplary use case describes one example of a typical userflow pattern. It is intended solely for explanatory purposes and not inlimitation. The user may be shopping in a retail store for a variety ofproducts. While moving through the store, the user may use an augmentedreality headset device (e.g., first computing device 120) to view anaugmented environment that includes virtual additions such as virtualprice tags, directional graphics, product information, or lists, such asa shopping list or a shopping cart. That is, the user's wearable deviceor mobile device may provide environmental data to the system (e.g., toservice provider terminal 110) and, in response, the system may augmentthe environmental data by adding virtual environmental data and providethe virtual environmental data to the user's computing device. Thisaugmented environment may be populated by pulling in synced user profileor device data, or through visual identification of products in the realenvironment. A second user can join in the experience by connecting withtheir device (e.g., second computing device 130). That is, the systemmay provide the environmental data and the virtual environmental data tothe second user's device to create a virtual environment. Using theirdevice, the second user can manipulate virtual items in the augmentedenvironment visible to both users. That is, the system may modify thevirtual environment on the second user's device and the augmentedenvironment on the first user's wearable device or mobile device todisplay enhanced content based on user input from the second user'sdevice. For example, the second user can select an item on their deviceby tapping on a virtual price tag and placing it into a virtual shoppingcart or a virtual fitting room. The first user can see this in theaugmented environment via their wearable device or mobile device, andwalk over to the selected item and go into an actual fitting room andtry it on. While trying on the item, the second user can see how it fitsand can further modify the virtual and augmented environments, forexample by making a swiping gesture to select different virtual colorsor virtual sizes for the item being tried on.

Another use case would be for providing virtual customer service. Ashopper can enter a store, be provided with an augmented reality device,and immediately be provided a virtual personal shopper to help themselect items. Alternatively, an employee at the merchant location or athird party may serve as a surrogate shopper for a user who is notpresent at the merchant location (e.g., to provide a shopping experiencethat would be otherwise unavailable or inconvenient due to location,physical handicap, or time constraint) or at another area at themerchant location (e.g., to allow a mom to continue shopping with herchild in the video game area while she waits in the customer servicearea). Since items in the augmented environment can actually bemanipulated by both users, the system provides a much more collaborativeand immersive experience than basic video chat applications. It iscontemplated that the interaction over system 100 can take placereal-time or be delayed.

Other exemplary use cases involve interactions between users each havingaugmented reality devices as described in the systems and methodsherein. Additionally, other exemplary use cases involve interactionsbetween users each having virtual reality devices as described in thesystems and methods herein.

Certain implementations of the disclosed technology are described abovewith reference to block and flow diagrams of systems and methods and/orcomputer program products according to example implementations of thedisclosed technology. It will be understood that one or more blocks ofthe block diagrams and flow diagrams, and combinations of blocks in theblock diagrams and flow diagrams, respectively, can be implemented bycomputer-executable program instructions. Likewise, some blocks of theblock diagrams and flow diagrams may not necessarily need to beperformed in the order presented, may be repeated, or may notnecessarily need to be performed at all, according to someimplementations of the disclosed technology.

These computer-executable program instructions may be loaded onto ageneral-purpose computer, a special-purpose computer, a processor, orother programmable data processing apparatus to produce a particularmachine, such that the instructions that execute on the computer,processor, or other programmable data processing apparatus create meansfor implementing one or more functions specified in the flow diagramblock or blocks. These computer program instructions may also be storedin a computer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meansthat implement one or more functions specified in the flow diagram blockor blocks. As an example, implementations of the disclosed technologymay provide for a computer program product, including a computer-usablemedium having a computer-readable program code or program instructionsembodied therein, said computer-readable program code adapted to beexecuted to implement one or more functions specified in the flowdiagram block or blocks. Likewise, the computer program instructions maybe loaded onto a computer or other programmable data processingapparatus to cause a series of operational elements or steps to beperformed on the computer or other programmable apparatus to produce acomputer-implemented process such that the instructions that execute onthe computer or other programmable apparatus provide elements or stepsfor implementing the functions specified in the flow diagram block orblocks.

As used herein, unless otherwise specified the use of the ordinaladjectives “first,” “second,” “third,” etc., to describe a commonobject, merely indicate that different instances of like objects arebeing referred to, and are not intended to imply that the objects sodescribed must be in a given sequence, either temporally, spatially, inranking, or in any other manner.

What is claimed is:
 1. A system for providing an interactive augmentedenvironment, the system comprising: one or more environmental datasensors configured to obtain environmental data comprising audio and/orvisual information representing a physical environment; an input/outputdevice configured to: transmit the environmental data to a firstcomputing device, receive, from the first computing device, virtualenvironmental data comprising data associated with a merchant, andreceive, from the first computing device, user input data associatedwith one or more predefined user inputs; an augmented environmentprocessor configured to: generate, based on at least some of theenvironmental data and at least some of the virtual environmental data,a signal comprising augmented environmental data, and modify the signalto integrate enhanced content based on the user input data received fromthe first computing device, the enhanced content being unrepresented inthe physical environment; and an augmented environmental data displayconfigured to receive the signal and display the augmented environmentaldata in accordance with the signal.
 2. The system of claim 1 wherein theenvironmental data and the virtual environmental data are transmitted toa second computing device to create a virtual environment.
 3. The systemof claim 1, wherein the augmented environment processor identifies oneor more products in the visual information.
 4. The system of claim 1,wherein the first computing device is configured to communicate with asecond computing device in real time.
 5. A system for providing aninteractive augmented environment, the system comprising: one or moreprocessors; and memory comprising instructions that, when executed bythe one or more processors, cause the system to: receive, from a firstcomputing device, first environmental data comprising first audio andfirst visual information representing a physical environment; identify,based on the first environmental data, a perceived environmentalindicator indicative of a virtual location corresponding to a physicallocation in the physical environment; augment the first environmentaldata by adding, based on the perceived environmental indicator, virtualenvironmental data associated with the perceived environmental indicatorand comprising data associated with a merchant and indicator displaydata corresponding to the perceived environmental indicator, theindicator display data being unrepresented in the physical environment;transmit, to the first computing device, the virtual environmental datato create an augmented environment that combines the virtualenvironmental data with the first environmental data; receive, from asecond computing device, second environmental data comprising secondaudio and second visual information representing the physicalenvironment; identify, based on the second environmental data, theperceived environmental indicator; transmit, to the second computingdevice, the virtual environmental data associated with the perceivedenvironmental indicator to create an augmented environment combining thevirtual environmental data with the second environmental data; receive,from the second computing device, user input data comprising aninstruction to modify the indicator display data; modify, based on theuser input data, the indicator display data to form modified virtualenvironmental data; receive, from the first computing device, thirdenvironmental data comprising third audio and third visual informationrepresenting the physical environment; identify, based on the thirdenvironmental data, the perceived environmental indicator; and transmit,to the first computing device, the modified virtual environmental datato update the augmented environment to display the modified indicatordisplay data.
 6. The system of claim 5, wherein identifying theperceived environmental indicator comprises receiving, from the firstcomputing device, a request to place a virtual marker at the virtuallocation corresponding to the physical location in the physicalenvironment.
 7. The system of claim 5, wherein the perceivedenvironmental indicator corresponds to a physical indicator present inthe physical environment.
 8. The system of claim 7, wherein the physicalindicator is associated with a physical product for purchase.
 9. Thesystem of claim 5, wherein the physical environment is a merchantlocation associated with the merchant.
 10. The system of claim 5,wherein the virtual environmental data includes one or more of productprice information, product inventory information, product manufacturinginformation, product marketing information or description, productratings or reviews, social information, related products, availablediscounts, and aesthetic visual effects.
 11. The system of claim 5,wherein the first and second computing devices each comprise one or moreof a mobile phone, a virtual reality headset, an augmented realityheadset, a wearable device, an internet of things (IoT) device, a tabletcomputer, and a personal computer.
 12. The system of claim 5, whereinaugmenting the first environmental data further comprises identifyingone or more physical objects in the first visual information.
 13. Thesystem of claim 12, wherein augmenting the first environmental datafurther comprises displaying information from the merchant thatcorresponds to at least one of the one or more physical productsidentified in the first visual information.
 14. The system of claim 5,wherein the user input data is associated with one or more predefineduser inputs.
 15. The system of claim 14, wherein the one or morepredefined user inputs comprises a selection of a product displayed inthe second virtual environment.
 16. The system of claim 14, wherein theone or more predefined user inputs comprises a modification of a productdisplayed in the second virtual environment.
 17. The system of claim 16,wherein the modification of a product displayed in the virtualenvironment comprises one or more of a color change, a size change, anda style change.
 18. The system of claim 16, wherein the instructions,when executed by the one or more processors, further cause the systemto: receive, from the first computing device or the second computingdevice, a purchase request to purchase a product associated with theperceived environmental indicator; transmit, to a financial serviceprovider, a purchase authorization request corresponding to the purchaserequest to purchase the product associated with the perceivedenvironmental indicator; and responsive to receiving authorization fromthe financial service provider, transmit the purchase request to themerchant.
 19. A method of providing an interactive augmentedenvironment, the method comprising: receiving, by a processor and from afirst computing device, first environmental data comprising first visualinformation representing a physical environment; responsive toidentifying, by the processor, one or more physical objects in the firstvisual information, augmenting, by the processor, the firstenvironmental data by adding virtual environmental data comprising dataassociated with a merchant; transmitting, by the processor and to thefirst computing device, the virtual environmental data to create a firstaugmented environment overlaying the virtual environmental data over afirst view of the physical environment; receiving, by the processor andfrom a second computing device, second environmental data comprisingsecond visual information representing the physical environment;responsive to identifying, by the processor, the one or more physicalobjects in the second visual information, transmitting, by the processorand to the second computing device, the virtual environmental data tocreate a second augmented environment overlaying the virtualenvironmental data over a second view of the physical environment;receiving, by the processor and from the second computing device, userinput data associated with one or more predefined user inputs, the userinput data comprising an instruction to modify a product displayed inthe virtual environment; modifying, by the processor and based on theuser input data, a visual indicator representing the product, the visualindicator being included in the virtual environment data and beingunrepresented in the physical environment; and transmitting, by theprocessor and to the first computing device, the modified virtualenvironmental data to update the augmented environment to display themodified visual indicator representing the product; and transmitting, bythe processor to the second computing device, the modified virtualenvironmental data to update the virtual environment to display themodified visual indicator representing the product.
 20. The method ofclaim 19 further comprising: receiving, from the first computing deviceor the second computing device, a purchase request to purchase theproduct; transmitting, to a financial service provider, a purchaseauthorization request corresponding to the purchase request to purchasethe product; and responsive to receiving authorization from thefinancial service provider, transmitting the purchase request to themerchant.